cacIRT (version 1.4)

Nonparametric Approach to CA and CC: Computes classification accuracy and consistency using Lathrop and Cheng's (2014) nonparametric approach.

Description

Computes classification accuracy and consistency with Lathrop & Cheng's (2014) approach. First, the kernel-smoothed estimate of the probability of a correct response, conditional on observed total score, is found with pnr(). Then, the method proceeds similar to class.Lee(). Using the nonparametric approach does not require a parametric IRT model, keeps the problem on the total score scale, and can produce more accurate CA and CC estimates when the IRT model's assumptions are violated (see Lathrop & Cheng, 2014).

Usage

Lee.pnr(cutscore, pnr.out)
pnr(resp, bw.g = NULL, alpha = .5)

Arguments

cutscore

A scalar or vector of cut scores on the total score scale. Should not include 0 or the max total score, as the end points are added internally.

pnr.out

The output from pnr(). It is a list of length 3 where pnr.out[[1]] is a vector of T evaluation points on the total score scale (integers from 0 to the max total score) pnr.out[[2]] is a vector of the observed d

resp

The response data matrix with rows as subjects and columns as items. Because the method is based on total score, the method is not robust to missing data. Any NA in resp will propogate to the output.

bw.g

The global bandwidth parameter. The default of NULL will estimate the global bandwidth with the optimal (in terms of MSE) estimate of the bandwidth for normally distributed variables. The default is generally a good starting point.

alpha

The adaptivity of the bandwidth parameter. A value of 0 means no adaptation and each evaluation point uses the value in bw.g. For, other values (up to and including 1), the bandwidth parameter will shrink if the evaluation point is in an area

Value

• Marginal
  A matrix with two columns of marginal accuracy and consistency per cut score (and simultaneous if multiple cutscores are given)
• Conditional
  A list of two matrixes, one for conditional accuracy and one for conditional consistency. Each matrix has one row per evaluation point.

References

Lathrop, Q. N., & Cheng, Y. (2014). A Nonparametric Approach to Estimate Classification Accuracy and Consistency. Journal of Educational Measurement, 51(3), 318-334. Lee, W. (2010) Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47, 1-17. Ramsay, J. O. (1991). Kernel Smoothing Approaches to Item Characteristic Curve Estimation. Psychometrika, 56(4), 611-630.

Examples

#Simulate simple response data

params <- matrix(c(1,1,1,1,-2,1,0,1,0,0,0,0),4,3)
theta <- rnorm(100)
rdm <- sim(params, theta)

pnr.out <- pnr(rdm)

resultsNP <- Lee.pnr(3, pnr.out)